{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Voice enabled room control system\n",
    "   \n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Aim\n",
    "\n",
    "* This application notebook shows working of a voice based room control system for controlling the ambience of a room.   \n",
    "  It uses an online service to recognize the given speech based commands to control various sensors and responds back to the user with an appropriate reply.\n",
    "\n",
    "\n",
    "## References \n",
    "* [Grove temperature](https://www.seeedstudio.com/Grove-Temperature-Sensor.html) \n",
    "* [Grove Servo](https://wiki.seeedstudio.com/Grove-Servo.html)  \n",
    "* [Grove Relay](https://www.seeedstudio.com/Grove-Relay.html)  \n",
    "* [Grove I2C OLED](https://wiki.seeedstudio.com/Grove-OLED_Display_0.96inch/) \n",
    "* [Grove RGB LED Stick](https://www.seeedstudio.com/Grove-RGB-LED-Stick-10-WS2813-Mini.html) \n",
    "* [Grove Base Shield V2.0](https://www.seeedstudio.com/Base-Shield-V2.html)   \n",
    "* [Speech Recognition](https://pypi.org/project/SpeechRecognition/)\n",
    "* [Text to Speech](https://gtts.readthedocs.io/en/latest/)\n",
    "\n",
    "## Last revised\n",
    "* 07 April 2021\n",
    "    + Initial version\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-box alert-warning\">\n",
    "   <h4 class=\"alert-heading\">Notes before running notebook </h4>\n",
    "    <ul>\n",
    "        <li> Please run the notebook while connected to regular wall power. As many modules are used in the application the USB power may not be sufficient</li>\n",
    "        <li> Connect to Internet (wired or wireless) before running the code cells as we use free online services</li>\n",
    "        <li> You will need Earphones+Mic to give voice commands to the application</li>\n",
    "    </ul> \n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "\n",
       "try {\n",
       "require(['notebook/js/codecell'], function(codecell) {\n",
       "  codecell.CodeCell.options_default.highlight_modes[\n",
       "      'magic_text/x-csrc'] = {'reg':[/^%%microblaze/]};\n",
       "  Jupyter.notebook.events.one('kernel_ready.Kernel', function(){\n",
       "      Jupyter.notebook.get_cells().map(function(cell){\n",
       "          if (cell.cell_type == 'code'){ cell.auto_highlight(); } }) ;\n",
       "  });\n",
       "});\n",
       "} catch (e) {};\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/javascript": [
       "\n",
       "try {\n",
       "require(['notebook/js/codecell'], function(codecell) {\n",
       "  codecell.CodeCell.options_default.highlight_modes[\n",
       "      'magic_text/x-csrc'] = {'reg':[/^%%pybind11/]};\n",
       "  Jupyter.notebook.events.one('kernel_ready.Kernel', function(){\n",
       "      Jupyter.notebook.get_cells().map(function(cell){\n",
       "          if (cell.cell_type == 'code'){ cell.auto_highlight(); } }) ;\n",
       "  });\n",
       "});\n",
       "} catch (e) {};\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import os\n",
    "import re\n",
    "from time import sleep\n",
    "\n",
    "from IPython.display import display, HTML, Audio, clear_output\n",
    "from pynq.overlays.base import BaseOverlay\n",
    "from pynq_peripherals import ArduinoSEEEDGroveAdapter, PmodGroveAdapter"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-box alert-info\">\n",
    "   <h4 class=\"alert-heading\">Library Dependencies </h4>\n",
    "Before proceeding the following libraries must be installed\n",
    "    <ul>\n",
    "        <li> speech_recognition </li>\n",
    "        <li> flac</li>\n",
    "        <li> portaudio19-dev</li>\n",
    "        <li> pyaudio</li>\n",
    "        <li> gtts </li>\n",
    "    </ul>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    import speech_recognition as sr\n",
    "except ImportError:\n",
    "    print(\"This script requires the the following module installs: \\n\"\n",
    "          \"1. FLAC : sudo apt-get install flac \\n\"\n",
    "          \"2. portaudio19-dev : sudo apt-get install portaudio19-dev \\n\"\n",
    "          \"3. pyaudio : sudo pip3 install pyaudio\\n\")\n",
    "    print(\"Then install speech_recognition module with: \\n\"\n",
    "         \"sudo pip3 install SpeechRecognition\")\n",
    "try:\n",
    "    from gtts import gTTS\n",
    "except ImportError:\n",
    "    print(\"This script requires the gtts module \\n Install with: sudo \"\n",
    "         \"pip3 install gtts\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Constructing application with Grove Base Shield V2.0 (Arduino)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Library compilation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "base = BaseOverlay(\"base.bit\")\n",
    "r = sr.Recognizer()\n",
    "\n",
    "pAudio = base.audio\n",
    "pAudio.set_volume(20)\n",
    "pAudio.select_microphone()\n",
    "pAudio.bypass(seconds=5)\n",
    "\n",
    "adapter = ArduinoSEEEDGroveAdapter(base.ARDUINO, D6='grove_relay',\n",
    "                                   I2C='grove_oled', A1='grove_temperature',\n",
    "                                   D7='grove_led_stick', D5='grove_servo')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create all the Grove devices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "relay = adapter.D6\n",
    "oled = adapter.I2C\n",
    "temp_sensor = adapter.A1\n",
    "led_stick = adapter.D7\n",
    "servo = adapter.D5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-box alert-danger\">\n",
    "    <h4 class=\"alert-heading\">Stop before proceeding further! </h4>\n",
    "     Please make sure you had connected all the sensors before switching the board ON. If the sensors are connected while the board is ON it can cause the system to shutdown. For safe use, you can note down the connections shown below, switch the board OFF make the physical connections and the switch the board back ON.\n",
    "    \n",
    "   <h4 class=\"alert-heading\">Make Physical Connections </h4>\n",
    "    <ul>\n",
    "        <li>Insert the Grove Base Shield into the Arduino connector on the board.</li>\n",
    "        <li>Connect the grove_temperature module to A1 connector of the Grove Base Shield.</li>\n",
    "        <li>Connect the grove_servo module to D5 connector of the Grove Base Shield.</li>\n",
    "        <li>Connect the grove_led_stick module to D7 connector of the Grove Base Shield.</li>\n",
    "        <li>Connect the grove_oled module to one of the I2C connector of the Grove Base Shield.</li>\n",
    "        <li> Connect the grove_relay module to D6 connector of the Grove Base Shield.</li></ul>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](data/room.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Construct the voice enabled room control system"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "global sensor_state\n",
    "\n",
    "oled.set_default_config()\n",
    "oled.set_normal_display()\n",
    "oled.set_next_row_wrap_mode()\n",
    "led_stick.clear()\n",
    "\n",
    "led_colors = [0xFF0000, 0xFF2300, 0xFF5700, 0xFF8C00, 0xFFC100, 0xFFF600,\n",
    "              0xD4FF00, 0x9FFF00, 0x6AFF00, 0x00FF00]\n",
    "\n",
    "sensor_state = {'relay': 'off', 'servo': 'off'}\n",
    "\n",
    "\n",
    "def t2s(text):\n",
    "    try:\n",
    "        tts = gTTS(text=text, lang='en', slow=False)\n",
    "        speech = os.getcwd() + '/output.mp3'\n",
    "        tts.save(speech)  # save a wav file\n",
    "        return speech\n",
    "    except:\n",
    "        return (\"data/check_internet.mp3\")\n",
    "\n",
    "\n",
    "def html_display(input_text):\n",
    "    return f\"\"\"\n",
    "            <h2><b style=\"color:#00008B\"> {input_text}</b></h2>\n",
    "            <br>\n",
    "           \"\"\"\n",
    "\n",
    "\n",
    "def oled_display(input_text):\n",
    "    oled.set_position(0, 0)\n",
    "    oled.put_string(\"I heard you say: \")\n",
    "    oled.set_position(3, 0)\n",
    "    oled.put_string(f\"{input_text}\")\n",
    "\n",
    "\n",
    "def reply(input_text):\n",
    "    display(Audio(t2s(input_text), autoplay=True))\n",
    "    sleep(3)\n",
    "\n",
    "\n",
    "def temperature():\n",
    "    return f'temperature is {temp_sensor.get_temperature():.2f} degree centigrade'\n",
    "\n",
    "\n",
    "def lights(text_in):\n",
    "    if 'on' in text_in and 'mood' not in text_in:\n",
    "        if sensor_state['relay'] == 'off':\n",
    "            relay.on()\n",
    "            sensor_state.update(relay='on')\n",
    "            return 'ok, turning the lights on'\n",
    "        else:\n",
    "            return 'lights are already on'\n",
    "    elif 'off' in text_in and 'mood' not in text_in:\n",
    "        if sensor_state['relay'] == 'on':\n",
    "            relay.off()\n",
    "            sensor_state.update(relay='off')\n",
    "            return 'ok, turning the lights off'\n",
    "    else:\n",
    "        return 'lights are already off'\n",
    "\n",
    "\n",
    "def mood(text_in):\n",
    "    if 'happy' in text_in or 'Happy' in text_in:\n",
    "        led_stick.clear()\n",
    "        for i in range(10):\n",
    "            led_stick.set_pixel(i, led_colors[i])\n",
    "            led_stick.show()\n",
    "        return 'I am glad you are having a good day. enjoy'\n",
    "    elif 'calm' in text_in or 'Calm' in text_in:\n",
    "        for i in range(10):\n",
    "            led_stick.set_pixel(i, 0x00FFFF)\n",
    "            led_stick.show()\n",
    "        return 'being calm is good. hope it gets better',\n",
    "    elif 'off' in text_in:\n",
    "        led_stick.clear()\n",
    "        return 'ok, turning the mood lights off',\n",
    "    else:\n",
    "        led_stick.clear()\n",
    "        return 'please let me know if your in a happy or a calm mood'\n",
    "\n",
    "\n",
    "def blinds(text_in):\n",
    "    if 'open' in text_in:\n",
    "        for x in range(10, 170, 10):\n",
    "            servo.set_angular_position(x)\n",
    "            sleep(0.1)\n",
    "        return 'I opened the blinds for you'\n",
    "    else:\n",
    "        for x in range(160, 0, -10):\n",
    "            servo.set_angular_position(x)\n",
    "            sleep(0.1)\n",
    "        return 'blinds are closed now'\n",
    "\n",
    "\n",
    "def exit_application():\n",
    "    clear_output()\n",
    "    led_stick.clear()\n",
    "    oled.clear_display()\n",
    "    return 'ok, goodbye'\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-box alert-info\">\n",
    "   <h4 class=\"alert-heading\">Notes before running the next cell </h4>\n",
    "    <p>You can speak the commands when the 4 LEDs on the board are lit up. Please use commands that are short and clear ~5 seconds for best interaction</p>\n",
    "    <b>Command list</b>\n",
    "    <ul>\n",
    "        <li> Light control</li>\n",
    "        <ul><li><b>'Switch/turn on Lights'</b></li></ul>\n",
    "        <ul><li><b>'Switch/turn off Lights'</b></li></ul>\n",
    "        <li> Temperature sensing</li>\n",
    "        <ul><li><b>What is the temperature?</b></li></ul>\n",
    "        <li> Blinds control</li>\n",
    "        <ul><li><b>Close the blinds please..</b></li></ul>\n",
    "        <ul><li><b>Open the blinds please</b></li></ul>\n",
    "        <li> Mood lighting</li>\n",
    "        <ul><li><b>I am in a Happy mood / Set mood to happy</b></li></ul>\n",
    "        <ul><li><b>I am in a Calm mood / Set mood to calm</b></li></ul>\n",
    "        <li> Exit</li>\n",
    "        <ul><li><b>Exit listening mode</b></li></ul>\n",
    "    </ul>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "count = 0\n",
    "relay.off()\n",
    "led_stick.clear()\n",
    "\n",
    "while True:\n",
    "    text_in = \"-------------\"\n",
    "    oled.clear_display()\n",
    "    hdisplay = display(HTML(''),\n",
    "                       display_id=True)  # Initialize display variable\n",
    "    [led.on() for led in base.leds]\n",
    "    hdisplay.update(HTML(html_display(\"Listening...\")))\n",
    "    pAudio.record(5)\n",
    "    pAudio.save(\"recording_1.wav\")\n",
    "    [led.off() for led in base.leds]\n",
    "    hdisplay.update(HTML(html_display(\"Processing..\")))\n",
    "\n",
    "    harvard = sr.AudioFile('recording_1.wav')\n",
    "    with harvard as source:\n",
    "        audio = r.record(source)\n",
    "\n",
    "    try:\n",
    "        text_in = r.recognize_google(audio)\n",
    "        count = 0\n",
    "    except:\n",
    "        count += 1\n",
    "        if count < 50:\n",
    "            clear_output()\n",
    "            continue\n",
    "        else:\n",
    "            reply('turning off due to timeout')\n",
    "            clear_output()\n",
    "            oled.clear_display()\n",
    "            relay.off()\n",
    "            led_stick.clear()\n",
    "            break\n",
    "\n",
    "    oled_display(text_in)\n",
    "\n",
    "    if re.search('weather|temperature', text_in):\n",
    "        temp_response = temperature()\n",
    "        reply(temp_response)\n",
    "    elif re.search('lights|light', text_in) and not re.search('mood', text_in):\n",
    "        light_response = lights(text_in)\n",
    "        reply(light_response)\n",
    "    elif re.search('mood|leds|RGB lights|RGB|happy|Happy|calm|Calm', text_in):\n",
    "        mood_response = mood(text_in)\n",
    "        reply(mood_response)\n",
    "    elif re.search('blind|blinds|Blind|Blinds', text_in):\n",
    "        blinds_response = blinds(text_in)\n",
    "        reply(blinds_response)\n",
    "    elif re.search('exit', text_in):\n",
    "        exit_response = exit_application()\n",
    "        reply(exit_response)\n",
    "        clear_output()\n",
    "        break\n",
    "    else:\n",
    "        reply(\"Sorry, I did not get that\")\n",
    "\n",
    "    sleep(4)\n",
    "    clear_output()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright (C) 2021 Xilinx, Inc\n",
    "\n",
    "SPDX-License-Identifier: BSD-3-Clause"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
